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SUMMARY 


At  the  Air  Force  Research  Laboratory  (AFRL)  in  Mesa,  AZ,  there  is  an  ongoing  program  of 
research  on  training  in  a  4-ship  F-16  Distributed  Mission  Training  (DMT)  system.  Typically,  a 
team  of  four  pilots  comes  to  the  laboratory  for  a  week-long  training  exercise.  They  fly  together  as  a 
4-ship  team  on  several  missions  designed  to  provide  exposure  to  a  range  of  combat  scenarios.  The 
missions  involve  extensive  briefing  and  debriefing  sessions  in  addition  to  the  time  in  the  simulators. 
In  order  to  track  the  effects  of  training,  several  inter-related  projects  are  underway  to  assess  the 
effectiveness  of  training  and  to  compare  different  training  methods.  In  the  project  reported  here,  we 
assessed  changes  in  the  ways  pilots  understand  important  concepts  related  to  the  training.  Pilots 
rated  the  relatedness  of  all  pairs  of  21  concepts  from  the  domain  of  air-to-air  engagements  both 
before  and  after  training.  Measures  of  the  internal  consistency  (Coherence)  of  the  ratings  and 
Pathfinder  networks  were  derived  from  the  ratings.  Data  from  a  group  of  the  most  experienced 
pilots  (experts)  provided  a  point  of  reference  for  the  less-experienced  pilots.  At  the  beginning  of  the 
week,  Coherence  was  significantly  correlated  with  previous  experience  in  fighter  aircraft  suggesting 
that  providing  consistent  ratings  depends  on  having  a  well- developed  mental  model  of  the  domain. 
Also,  there  was  a  significant  correlation  between  experience  level  and  similarity  to  the  expert 
reference  group  at  the  beginning  of  the  week  supporting  the  general  validity  of  the  measurement 
methods.  There  was  a  significant  negative  correlation  between  experience  level  and  change  in 
similarity  to  experts  from  pre-  to  post  training  ratings.  Greater  changes  were  found  for  the  least- 
experienced  pilots.  As  a  result  of  these  changes,  correlations  with  prior  experience  level  were  no 
longer  statistically  significant  at  the  end  of  the  week.  Further  analyses  on  a  group  of  the  least- 
experience  pilots  (novices)  lead  to  similar  conclusions.  In  particular,  there  was  a  significant 
difference  in  mean  Coherence  between  the  experts  and  novices  at  the  beginning  of  the  week  but  not 
at  the  end  of  the  week.  Also  novices  showed  significant  pre-  to  post  training  increases  in  Coherence 
and  in  similarity  to  experts.  These  measurement  methods  appear  to  provide  a  basis  for  evaluating 
conceptual  change.  These  assessment  methods  should  prove  to  be  a  useful  adjunct  to  performance- 
based  methods  of  assessing  training. 
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INTRODUCTION 
F-16  DMT  Training 

At  the  Air  Force  Research  Laboratory  (AFRL)  in 
Mesa,  AZ,  there  is  an  ongoing  program  of 
research  on  training  in  a  4ship  F-16  Distributed 
Mission  Training  (DMT)  system.  Typically,  a  team 
of  four  pilots  comes  to  the  laboratory  for  a  week- 
long  training  exercise.  They  fly  together  as  a  4 
ship  team  on  several  missions  designed  to  provide 
exposure  to  a  range  of  combat  scenarios.  The 
missions  involve  extensive  briefing  and  debriefing 
sessions  in  addition  to  the  time  in  the  simulators. 
The  simulators  at  AFRL  can  be  connected  in  real¬ 
time  to  simulators  at  other  locations  in  the  United 
States  and  to  stations  involving  other  roles  in 
addition  to  fighter  pilots  such  as  AWACS 
operators.  Thus,  a  variety  of  training  settings  can 
be  devised  in  the  overall  DMT  system.  Several 
research  projects  are  focused  on  designing  and 
evaluating  training.  In  this  paper,  we  discuss  one 
line  of  research  on  evaluating  fighter-pilot  training. 

Assessment  of  Training 

There  are  many  reasons  to  be  concerned  with 
evaluating  the  training  provided  by  this  system.  Of 
primary  importance  is  to  determine  that  the 
objectives  of  the  training  are  being  met.  There  is 
also  concern  with  developing  training  methods  that 
will  produce  the  largest  gains  possible  from  the 
week  of  training.  Numerous  research  questions 
can  be  framed  concerning  performance  in  the 
simulators,  performance  in  actual  flight,  and  the 
acquisition  of  requisite  knowledge. 

Simulator  performance.  Perhaps  the  most  direct 
approach  to  evaluating  the  impact  of  training  is  to 
assess  performance  in  the  simulated  missions 
encountered  as  part  of  training.  Such  efforts  are 
underway  in  the  lab,  but  there  are  some  problems 
associated  with  using  performance  in  the 
simulators  as  the  basis  for  evaluating  progress  in 
training.  First,  because  the  complexity  of  the 
missions  typically  increases  dramatically  over  the 
course  of  the  week,  direct  comparisons  of 
performance  are  suspect.  Using  benchmark 
missions  has  been  explored,  but  finding  an 
appropriate  level  of  difficulty  for  such  missions  is 
problematic  because  using  more  complex 
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missions  early  in  the  week  may  place  unrealistic 
demands  on  pilots  who  need  additional 
experience.  Conversely,  easier  missions  may  not 
be  sufficiently  challenging  to  show  deficiencies 
even  earlier  in  the  week. 

Performance  in  missions  or  further  training. 

Ultimately,  training  should  have  a  positive  impact 
on  performance  in  actual  missions  flown  and  on 
the  progress  of  further  training  in  operational  units. 
Efforts  are  also  underway  to  gather  and  analyze 
data  that  bear  on  the  transfer  of  training  to  actual 
flying  performance  and  on  performance  in  further 
training.  Such  data  take  considerable  time  and 
effort  to  collect,  and  there  are  major  issues 
associated  with  developing  appropriate  methods 
and  measurements  to  assess  performance  in 
flight.  In  this  project,  we  concentrated  on 
evaluating  pilots’  acquisition  of  the  knowledge 
required  to  understand  combat  situations  and  to 
make  good  decisions  about  possible  courses  of 
action. 

Knowledge.  Still  another  approach  to  evaluating 
the  impact  of  training  is  to  examine  changes  in 
knowledge  as  a  function  of  training.  Clearly,  the 
ultimate  goal  of  pilot  training  is  to  produce 
expertise  in  performance  in  flight.  However,  it  is 
also  clear  that  expert  performance  depends  on  the 
acquisition  of  both  knowledge  and  skill.  Assessing 
skill  requires  observation  in  live  performance,  but 
assessing  knowledge  can  be  done  outside  the  live 
performance  environment.  This  independence  of 
assessment  from  the  training  allows  greater 
control  over  the  collection  of  data  pertinent  to 
assessing  knowledge.  In  this  investigation,  we 
choose  to  approach  knowledge  by  means  of  how 
pilots  see  the  relationships  among  various 
concepts  in  the  air-to-air  combat  maneuvering 
domain.  This  area  receives  considerable  attention 
in  the  simulator  training  so  it  is  natural  to  suppose 
that  there  should  be  an  impact  on  pilot’s 
understanding  of  these  concepts. 

The  technique  we  used  is  generally  known  as 
Pathfinder  Network  Scaling  (see  Schvaneveldt, 
Dearholt,  &  Durso,  1989;  Schvaneveldt,  Durso,  & 
Dearholt,  1989;  Schvaneveldt,  1990).  The  method 
uses  individuals'  judgments  as  a  source  from 
which  to  extract  underlying  network  structures. 
Among  various  applications,  this  method  has  been 
used  to  capture  expert-novice  differences  in 
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conceptual  structure  (USAF  pilots  -  Schvaneveldt, 
et  al.,  1985;  also  see  Cooke  &  Schvaneveldt,  1988 
and  Rowe,  Cooke,  Hall  &  Halgren,  1996),  to 
assess  student  knowledge  acquisition  (Goldsmith, 
Johnson,  &  Acton,  1991  and  to  analyze  and 
design  user-system  interfaces  (McDonald  & 
Schvaneveldt,  1988).  Pathfinder  networks  stem 
from  research  and  theory  in  cognitive  psychology, 
and  they  represent  one  example  of  a  bridge 
between  basic  research  and  application  (see 
Gillan  &  Schvaneveldt,  1999). 

Graph  Theory  and  Networks 

The  abstract  model  underlying  the  network  model 
of  relationships  among  concepts  is  graph  and 
network  structures.  In  mathematical  graph  theory, 
a  graph  is  an  abstraction  consisting  of  a  set  of 
nodes  and  a  set  of  pairs  of  the  nodes  (Harary, 


1969).  Each  such  pair  of  nodes  is  called  a  link. 
The  links  can  be  directed  or  undirected.  The 
nodes  represent  some  entities  (concepts  in  this 
case.)  and  the  links  represent  relationships 
between  nodes. 

A  network  is  a  graph  with  weights  (or  costs) 
associated  with  the  links.  An  example  of  a 
network  using  familiar  concepts  from  aviation  is 
shown  in  Figure  1  (from  Schvaneveldt,  et.  al, 
2000).  We  often  expand  these  basic  definitions  to 
include  distinguishing  different  types  of  nodes  and 
different  types  of  links.  Taken  as  a  whole  the 
collection  of  nodes  and  links  can  represent  how 
individuals  or  groups  of  individuals  view  the 
relationships  among  concepts.  As  in  previous 
research,  we  hypothesize  that  differences  in 
expertise  will  be  reflected  by  differences  in  link 
structures. 


Traffic! 


Figure  1.  A  Pathfinder  Network  for  Selected  Aviation  Concepts. 


2 


METHOD 

Participants 

In  all,  rating  data  were  obtained  from  84  Air  Force 
pilots.  These  pilots  came  from  several  different 
operational  units  around  the  United  States.  They 
varied  greatly  in  experience,  and  they  came  to  the 
laboratory  with  different  objectives  in  mind. 
Several  analyses  of  the  data  were  based  on  all  of 
these  pilots.  Some  focused  analyses  were 
performed  on  two  groups  of  pilots,  a  group  of  six 
pilots  with  the  highest  levels  of  experience  (more 
than  1900  hours  of  flying  fighter  aircraft)  and 
another  group  of  eight  with  the  least  experience 
(80-105  hours  of  flying  fighter  aircraft). 

Concept  Selection 

The  concepts  rated  by  the  pilots  were  selected 
from  advanced  air-to-air  combat  maneuvering 
scenarios  (see  Table  1). 


Table  1.  Concepts  Rated  for  Relatedness 


Alamo 

AMRAMM 

Bandit/Hostile 

Beam  deploy 

BVR 

Doppler  pick 

Factor  bandit  range 

Grinder 

IRMD 

Launch  and  leave 

MOR 

Multiple  groups  in  azimuth 

Notch 

Multiple  groups  in  range 

PID 

Pit  Bull 

Preserve  range 

Real  world  ROE 

Targeting/Sorting 

Unknown  group 

Visual  mutual  support 

Concept  Rating 

Pathfinder  scaling  requires  measures  of  proximity 
for  all  pairs  of  elements  to  be  scaled.  In  this  study, 
ratings  were  based  on  210  judgments  of  the 
degree  of  relatedness  for  the  concepts  in  each 
pair  of  the  21  concepts.  From  such  data,  the 
Pathfinder  scaling  algorithm  produces  a  network 
showing  the  connections  between  the  various 
concepts. 

Variables 

In  this  section,  we  define  the  variables 
investigated  in  the  analyses  of  the  data. 

We  used  the  logarithm  of  the  number  of  hours 
flying  F-16  fighter  aircraft  as  the  measure  of 
Experience. 


Aside  from  measures  of  pilot  experience,  all  other 
variables  in  the  data  analysis  were  based  on  the 
pilots’  ratings  of  all  the  pairs  of  concepts.  All  84 
pilots  performed  the  rating  task  at  the  beginning  of 
the  week  (Rating  1),  but  only  60  completed  the 
ratings  at  the  end  of  the  week  ( Rating2 ).  A 
network,  PFNET(r  =  infinity,  q  =  2),  was  derived 
from  each  set  of  ratings.  The  networks  are 
referred  to  as  Netl  and  Net2  for  Ratingl  and 
Rating2,  respectively. 

Expert  reference  data  were  obtained  by  averaging 
the  ratings  given  by  the  six  most  experienced 
pilots.  We  refer  to  this  set  of  average  ratings  as 
AveExpRate.  Applying  the  Pathfinder  scaling 
algorithm  (r  =  infinity,  q  =  2)  to  AveExpRate 
produced  the  referent  expert  PFNET,  the  ExpNet. 
One  additional  analysis  of  rating  data  yields  a 
coherence  measure  which  assesses  the  internal 
consistency  of  the  ratings.  Coherence  is 
computed  in  two  steps.  First,  a  correlation  of 
ratings  for  each  pair  of  items  is  computed.  For 
example  for  items  1  and  2,  the  ratings  of  item  I 
across  all  of  the  other  items  is  correlated  with  item 
2  across  all  of  the  other  items.  This  determines 
the  extent  to  which  the  items  in  a  pair  are  rated 
similarly  against  the  other  items.  The  second  step 
correlates  these  correlations  with  the  ratings  given 
for  the  pairs.  This  can  be  seen  as  comparing  the 
direct  rating  of  a  pair  with  the  indirect  relatedness 
inferred  from  the  similarity  of  the  ratings  for  the 
items  in  each  pair. 

The  coherence  measure  has  been  shown  to 
reflect  levels  of  expertise  in  that  raters  with  more 
expertise  in  the  concept  domain  generally  produce 
higher  coherence  scores  compared  with  less 
experienced  raters.  Coherence  can  also  reveal 
unsystematic  ratings  that  might  be  provided  by  a 
participant  who  does  not  take  the  rating  task 
seriously  resulting  in  careless  ratings.  Given  that 
the  rating  task  requires  a  large  number  of 
judgments,  it  is  difficult  for  participants  to  be 
consistent  by  remembering  earlier  ratings.  Rather, 
consistency  more  likely  stems  from  using  a  clear 
understanding  of  the  domain  as  a  basis  of  the 
ratings.  Coherence  was  computed  from  both 
Ratingl  and  Rating2  yielding  Cohl  and  Coh2, 
respectively.  CohChange  is  Coh2-  Cohl. 

We  also  defined  variables  to  assess  the  extent  to 
which  the  ratings  and  the  networks  obtained  from 
individuals  are  similar  to  the  measures  obtained 
from  the  experts. 
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RatelSim  assesses  the  similarity  of  individual’s 
ratings  at  the  beginning  of  the  week  with  the 
average  of  the  experts’  ratings.  It  is  obtained  by 
correlating  AveExpRate  with  Ratingl  for  each 
pilot. 

Rate2Sim  assesses  the  similarity  of  individual’s 

ratings  at  the  end  with  Ynect2ngs.  1hance68  Tc180  0  -(Rate2  Tw  ( for  each)  Tj  0  Tc  080  0  -obtained  byj  0  To  0.2895  T )  Tj  0 
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Figure  2.  The  Pathfinder  Network  Derived  from  the  Average  of  Expert  Ratings. 


There  were  some  indications  in  these  data  that  the 
effects  we  were  observing  were  largely  due  to  the 
least  experienced  pilots  in  the  sample.  When 
these  8  pilots  were  removed  from  the  data  set,  all 
of  the  correlations  reported  above  became  non¬ 
significant.  We  decided  that  a  closer  look  at  the 
data  from  these  8  Novice  Pilots  was  in  order  (see 
Table  3). 

Novices 

It  is  interesting  to  note  that  all  of  the  changes  were 
significant  for  the  novices  as  is  shown  by  the  t- 
tests  at  the  bottom  of  the  table.  Novices  show 
significant  increases  in  Coherence  and  in  their 
similarity  to  experts  both  in  rating  similarity  and  in 
network  similarity. 

Not  shown  in  Table  3  is  a  comparison  of  the  rating 
coherence  between  these  novice  pilots  and  the 
group  of  six  experts.  For  the  novice  ratings  at  the 
beginning  of  the  training  ( Cohl ),  there  was  a 


significant  difference  between  the  novices  and  the 
experts  (Novice  Mean  =  0.34,  expert  Mean  =  0.58, 
t=  2.70,  df  =  12.  p=  0.019).  After  training,  the 
difference  is  not  significant.  For  these  differences 
to  appear  with  such  a  small  sample  size  is 
intriguing,  but  the  findings  should  be  replicated 
with  additional  observations. 

Overall,  it  appears  that  the  methods  used  in  the 
present  investigation  are  most  sensitive  to  the 
impact  of  training  on  the  least  experienced  pilots. 
Perhaps  it  is  not  surprising  that  these  pilots  should 
show  the  greatest  impact  of  the  training.  They 
certainly  have  a  greater  distance  to  go  in 
achieving  expert-level  knowledge  and 
performance. 
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Table  3.  Summary  of  Data  from  Novice  Pilots  (<100  hours) 


Coherence 

Similarity  of  Experts 
and  Novices 
Ratings 

Similarity  of  Experts 
and  Novices 
Networks 

Cohl 

Coh2 

Coh 

Change 

Ratel 

Sim 

Rate2 

Sim 

Rate 

Sim 

Change 

Netl 

Sim 

Net2 

Sim 

Net  Sim 
Change 

0.19 

0.44 

0.26 

0.32 

0.43 

0.11 

0.03 

0.04 

0.01 

0.44 

0.38 

-0.06 

0.56 

0.63 

0.07 

0.11 

0.15 

0.04 

0.36 

0.54 

0.18 

0.37 

0.63 

0.25 

0.07 

0.10 

0.04 

0.50 

0.69 

0.20 

0.50 

0.62 

0.12 

0.05 

0.05 

0.01 

0.59 

0.57 

-0.02 

0.51 

0.64 

0.14 

0.07 

0.11 

0.03 

0.22 

0.39 

0.17 

0.32 

0.52 

0.20 

0.05 

0.15 

0.10 

0.47 

0.54 

0.09 

-0.03 

0.33 

0.36 

-0.11 

0.48 

0.58 

-0.01 

0.13 

0.14 

0.34 

0.48 

0.15 

0.38 

0.56 

0.21 

0.06 

0.11 

0.05 

t 

2.78 

t 

3.17 

t 

2.78 

df 

6 

df 

6 

df 

6 

P 

0.032 

P 

0.019 

P 

0.032 

Teams  and  Similarity  Changes 

An  analysis  of  the  similarity  of  pilots  to  one 
another  was  devoted  to  determining  whether  the 
members  of  the  teams  of  four  pilots  tend  to 
become  more  similar  to  one  another  over  the 
course  of  the  week's  training  in  the  simulators. 
While  the  data  show  that  the  ratings  of  team 
members  are  more  highly  correlated  at  the  end  of 
the  week  ( Rating2 ,  mean  r  =  .455)  than  they  are 
at  the  beginning  of  the  week  (Ratingl,  mean  r  = 
.393).  The  difference  (.062)  is  significant  ip  < 
.001).  Although  it  is  tempting  to  see  these  data  as 
supporting  development  of  common  mental 
models  among  team  members,  it  is  instructive  to 
examine  the  similarity  of  pilots  in  different  teams. 
A  very  similar  result  occurs  comparing  members 
of  different  teams.  Pilots  show  higher  correlations 
at  the  end  of  the  week  (mean  r=  .426)  than  at  the 
beginning  of  the  week  (mean  r=  .384).  This  .042 
difference  is  also  significant  (p  <  .001). 
Apparently  the  increase  in  similarity  should  be 
seen  as  an  effect  of  the  common  training 
environment  and  the  similarity  of  the  situations 
encountered  in  training  rather  than  changes  that 
are  peculiar  to  the  4-ship  teams. 

Identifying  Discriminating  Concepts 

A  final  analysis  was  focused  on  identifying  the 
concepts  that  are  viewed  most  distinctly  by  expert 
and  novice  pilots.  Ratings  of  novice  pilots  were 


averaged  [AveNovRate).  Then  the  differences 
between  AveNovRate  and  AveExpRate  were 
computed.  Summing  the  absolute  values  of  these 
differences  for  each  of  the  21  concepts  allows  us 
to  order  them  by  the  magnitude  of  the  differences 
between  expert  and  novice  ratings.  This  ordering 
is  shown  in  Table  4. 


Table  4.  Concepts  that  Discriminate 


Most  Discriminating 

Least  Discriminating 

Rank 

Concept 

Rank 

Concept 

1 

Grinder 

1 

Unknown  group 

2 

Pit  Bull 

2 

Alamo 

3 

MOR 

3 

Doppler  pick 

4 

Multiple 
groups  in 
range 

4 

Notch 

5 

Preserve 

range 

5 

AMRAMM 

6 

BVR 

6 

Multiple  groups 
in  azimuth 

7 

Beam  deploy 

7 

PID 

8 

Visual  mutual 
support 

8 

Real  world  ROE 

9 

Launch  and 
leave 

9 

Targeting/Sorting 

10 

IRMD 

11 

Factor  bandit 
range 

12 

Bandit/Hostile 

Knowing  about  the  relative  difficulty  of  these 
concepts  could  usefully  feed  back  into  the  design 
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of  training  by  showing  what  the  novices  seem  to 
know  the  least  about.  By  ensuring  that  these 
concepts  receive  adequate  coverage  in  the 
briefing  and  debriefing  sessions,  novices  may  be 
better  prepared  to  deal  with  the  corresponding 
aspects  of  the  simulator  flights. 

SUMMARY  AND  CONCLUSIONS 

Several  aspects  of  the  data  reported  here  showed 
that  a  week  of  training  in  the  DMT  system  at 
AFRL  produces  measurable  changes  in  the  way 
less  experienced  pilots  view  a  collection  of 
concepts  from  the  domain  of  air-to-air  combat. 
The  major  effects  were  found  for  the  least 
experienced  pilots  in  our  sample.  Perhaps  pilots 
with  more  experience  have  already  mastered  the 
understanding  of  the  domain  so  training  and 
practice  does  not  lead  to  major  changes.  It  is 
possible  that  another  set  of  more  advanced 
concepts  would  reveal  changes  in  more 
experienced  pilots,  and  there  may  be  other 
methods  of  assessment  that  should  be  pursued. 

We  are  developing  a  method  for  more  directly 
evaluating  knowledge  associated  with  assessing 
combat  situations  and  selecting  appropriate 
courses  of  action.  Watch  for  further 
developments. 

Although  there  were  overall  increases  in  similarity 
among  pilots  after  training,  there  was  no  indication 
that  member  of  teams  become  more  similar  than 
do  members  of  different  teams.  The  increased 
similarity  is  apparently  due  to  pilots  being  exposed 
to  similar  scenarios  in  training  rather  than  to  pilots 
learning  more  about  members  of  pilots’  teams. 
Other  studies  of  team  cognition  (see  Cooke, 
Salas,  Cannon-Bowers,  &  Stout,  2000)  have  ofter 
looked  at  increasing  similarity  amont  team 
members.  The  data  reported  here  suggest  that  it 
may  be  useful  to  determine  the  extent  to  which 
increasing  similarity  is  due  to  exposure  to  similar 
task  conditions  as  opposed  to  training 
experiences  with  members  of  a  team. 

The  results  of  our  study  have  had  some  effects  on 
the  design  of  the  training  environment.  The 
concepts  we  identified  as  leading  to  the  greatest 
differences  between  novices  and  experts  have 
received  more  attention  in  the  briefing  and 
debriefing  sessions  accompanying  the  training 
sessions.  Other  ways  in  which  the  information  we 
collect  can  be  used  in  training  are  being 
considered.  For  example,  pilots  could  be 
presented  with  networks  like  the  one  shown  in 
Figure  2  to  elicit  discussion  about  the  nature  of 
the  relationships  depicted  by  the  links.  Inevitably 


such  discussion  draws  out  similarities  and 
differences  in  the  ways  pilots  think  about  the 
concepts.  A  hypothesis  that  could  be  investigated 
is  that  training  would  benefit  from  such 
discussions,  discussions  that  are  driven  by  pilots 
underlying  conception  of  air-to-air  combat. 

We  are  also  pursuing  ways  of  tapping  into  more 
procedural  knowledge  using  rating  and  network 
scaling  procedures.  There  is  much  more  to  be 
done. 
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